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Introduction 

Iln this chapter first the dynamics of the continuous space recurrent neural 
networks will be examined in a general framework. 

■ Then, the Hopfield Network as a special case of this kind of networks will be 
introduced. 
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2.1. Dynamical Systems 

■ The dynamics of a large class of neural network models, may be represented by 
a set of first order differential equations in the form 

j t X ] (t) = F ] (x (/)) j = l..N (2.1.1) 

where F. is a nonlinear function of its argument. 

• Ina more compact form it may be reformulated as 

^x(0 = F(x(0) (2.1.2) 

at 

where the nonlinear function F operates on elements of the State vector 
x(t) in an autonomous way, that is F(x(r)) does not depend explicitly on 
time t. 

i F(x) is a vector field in an N-dimensional State space. Such an equation is called 
State space equation and x(t) is called the State of the system at particular 
time t. 
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2.1. Dynamical Systems Existence and Uniqueness 

For the State space equation (2.1.2) to have a solution and for the solution to be 
unique, we have to impose certain restrictions on the vector function F (x(t)). 

i For a solution to exist, it is sufficient that F(x) to be continuous in all of its 
arguments. 

V For uniqueness of the solution, F(x) should satisfy Lipschitz condition. 

Let ||x|| denote a norm, which may be the Euclidean length, Hamming 
distance or any other one, depending on the purpose. Let x and y be a 
pair of vectors in an open set S, in vector space. Then according to the 
Lipschitz condition, there exists a constant k such that 

II F( x ) - F(y) || <*r||x-y|| (2.1.3) 

for all x and y in S. A vector F(x) that satisfies equation (2.1.3) is said to 
be Lipschitz. In particular, if all partial derivatives 3/p(x)/dx y . are finite 
everywhere, then the function F(x) satisfies the Lipschitz condition [Haykin 
94]. 
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2.1. Dynamical Systems Phase Space 

The phase space of a dynamical system describes the global characteristics of the 
motion rather than the detailed aspects of analytic or numeric solutions of the 
equation. 

I At a particular instant of time t, a single point in the n-dimensional phase space 
represents the observed State of the State vector, that is x(t). 

I Changes in the State of the system with time t are represented as a curve in the 
phase space, each point on the curve carrying (explicitly or implicitly) a 
label that records the time of observation. 

■ This curve is called a trajectory or orbit of the system. Figure 2.1.a. illustrates a 
trajectory in a two dimensional system. 


ANN ee543 

C rJ APTER JJ ; Recurrent Neural Networks 
2.1. Dynamical Systems Phase Space 

The family of trajectories each for a different initial condition x(0) is called the phase 
portrait of the system (Figure 2.1.b). 

i The phase portrait includes all those points in the phase space where the field 
vector F(x) is defined. 

I For an autonomous system, there will be one and only one trajectory passing 
through an initial State.. 


• The tangent vector, that is dx(t)/dt, represents the instantaneous velocity F(x(r)) 
of the trajectory. 


t 2 7 



( 


Figure 2.1. a) A two dimensional trajectory b) Phase portrait 
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2.3. Major forms of Dynamical Systems 

We distinguish three major forms dynamical system, for fixed weights and inputs 
(Figure 2.2): 



Figure 2.2. Three major forms of dynamical systems 
a) Convergent b) Oscillatory c) Chaotic 
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2.3. Major forms of Dynamical Systems 

a) Convergent: every trajectory x(t) converges to some fixed point, which is a 
State that does not change over time (Figure 2.2.a). 

l These fixed points are called the attractors of the system. 

i The set of initial States x(0) that evolves to a particular attractor is called the 
basin of attraction. 

• The locations of the attractors and the basin boundaries change as the 
dynamical system parameters change. For example, by altering the 
external inputs or connection weights in a recurrent neural network the 
basin attraction of the system can be adjusted. 



EE543 - ANN - CHAPTER 2 


4 




Ugur HALICI - METU EEE - ANKARA 


11/18/2004 


ANN ee543 

JJ i Recurrent Neural Networks 


2.3. Major forms of Dynamical Systems 

b) Oscillatory: every trajectory converges either to a cycle or 
to a fixed point. A cycle of period T satisfies 

x(t+T)=x(t) for all times t (Figure 2.2. b) 



b) Chaotic: most trajectories do not tend to cycles or fixed 
points. One of the characteristics of chaotic systems 

is that the long-term behavior of trajectories is extremely 
sensitive to initial conditions. That is, a slight change in the 
initial State x(0) can lead to very different behaviors, as t 
becomes large. (Figure 2. 2. c) 
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2. 4. Gradient, Conservative and Dissipative Systems Gradient 

V For a vector field F(x) on State space x(t) eR n , the V operator helps in formal 
description of the system. In faet, V is an operational vector defined as: 

A AJ- < 2A1 > 

l If the V operator applied on a scalar funetion E of vector x(t), that is 



is called the gradient of the funetion E and extends in the direction of the 
greatest rate of change of E and has that rate of change for its length. 
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2. 4. Gradient, Conservative and Dissipative Systems Level surfaces 

■ If we set E(x)=c, we obtain a family of surfaces known as level surfaces of E, 
as x takes on different values. On the assumption that E is single valued 
at each point, one and only one level surface passes through any given 
point P. The gradient of E(x) at any point P is perpendicular to the level 
surface of E, which passes through that point. (Figure 2.3) 
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2. 4. Gradient, Conservative and Dissipative Systems Divergence 

For a vector field 


F(x)-[Fi(i) F 2 (x) ... ^V(x)] T 


(2.4.3) 


the inner product 

V 7 c- dF x dF 2 dF N 

dx x dx 2 dx jy 

is called the divergence of F, and it has a scalar value. 


(2.4.4) 
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2. 4. Gradient, Conservative and Dissipative Systems 
Dissipative ve and conservative systems 

l Consider a region of volume V and surface S in the phase space of an 
autonomous system, and assume a flow of points from this region. 

l Let n denote a unit vector normal to the surface at dS pointing outward from the 
enclosed volume. 

I Then, according to the divergence theorem, the relation 

f (F(x).n )dS = [ (V.F (x))dV (2.4.5) 

^ s " v 

holds between the volume integral of the divergence of F(x) and the 
surface integral of the outwardly directed normal component of F(x). 
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2. 4. Gradient, Conservative and Dissipative Systems 
Dissipative ve and conservative systems 


f (F(x).n)dS=[ (V.F (x))dV (2.4.5) 

•'s * v 

• The quantity on the left-hand side of Eq. (2.4.5) is recognized as the net flux 
flowing out of the region surrounded by the closed surface S. If the 
quantity is zero (or equivalently V-F(x) = 0 in V), the system is 

conservative ; if it is negative (or VF(x)<0 in V), the system is 

dissipative. 

i If the system is dissipative, this guarantees the stability of the system. 
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2.5. Equilibrium States 

Remember 

^x(0 = F(x(0) (2A.2) 

at 


•A constant vector x* satisfying the condition 

F(x*) = 0 , 


(2.5.1) 


is called an equilibrium State (stationary State or fixed point) of the 
dynamical system defined by Eq. (2.1.2). 

• Since it results in 


dx L 
dt 


= 0 for i = 1..N , 
x* 


(2.5.2) 


the constant function x(t)=x* is a solution of the dynamical system. 
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2.5. Equilibrium States 

Remember 

■^-x(t) = F(x(t)) (2.1.2) 

at 

F(x*) = 0, (2.5.1) 

• If the system is operating at an equilibrium point, then the State vector stays 

constant, and the trajectory with an initial State x(0)=x* degenerates to a 
single point. 

• We are frequently interested in the behavior of the system around the equilibrium 

points, and try to investigate if the trajectories around the equilibrium 
points are converging to the equilibrium point, diverging from it or staying 
in an orbit around the point or combination of these. 

• The use of a linear approximation of the nonlinear function F(x) makes it easier to 
understand the behavior of the system around the equilibrium points. 
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2.5. Equilibrium States 

Remember 

~x(f) = F(x(f)) (2.1.2) 

at 

• Let x=x*+A x be a point around x*. If the nonlinear function F(x) is smooth and if 

the disturbance Ax is small enough then it can be approximated by the 
first two terms of its Taylor expansion around x* as: 

F(x * +Ax) = F(x*) + F'(x*)Ax (2.5.3) 

where 

F’(x*) = -Af| (2.5.4) 

3 X X=X* 

that is, in particular: 

= | . (2.5.5) 

& x i x=x* 

• Notice that F(x*) and F '(x*) in Eq. (2.5.3) are constant, therefore it is a linear 

equation in terms of A x. 
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2.5. Equilibrium States 

Remember 

~ x (0 = F(x(/)) (2.1.2) 

at 

F(x*) = 0 , (2.5.1) 

F(x*+Ax) = F(x*) + F'(x*)Ax (2.5.3) 

•Since an equilibrium point satisfies Eq. (2.5.1), we obtain 


F(x *+Ax) = F'(x*)Ax 

(2.5.6) 

On the other hånd, since 


— (x*+Ax) = — Ax 
dt dt 

(2.5.7) 

the Eq. (2.1.2) becomes 


— Ax = F'(x*)Ax 
dt K 7 

(2.5.8) 


ANN ee543 


C E APTER JJ ^ Recurrent Neural Networks 


2.5. Equilibrium States 

Remember 

— Ax = F'(x*)Ax (2.5.8) 

dt 


•Since Eq. (2.5.8) defines a homogenous differential equation with constant real 
coefficient, the eigenvalues of the matrix F ’(x*) determines the behavior 
of the system. 

• In order to have Ax(f) to diminish as f-^o o, we need the real parts of all the 
eigenvalues to be negative. 
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2. 6. Stability 

■ An equilibrium State x* of an autonomous nonlinear dynamical system is called 
stable, if for any given positive s, there exists a positive 5 satisfying, 

||x(0)-x*|| < 8 ||x(f)-x*|| < e forail t> 0. (2.6.1) 

1 If x* is a stable equilibrium point, it means that any trajectory described by the 
State vector x(t) of the system can be made to stay within a small 
neighborhood of the equilibrium State x* by choosing an initial State x(0) 
close enough to x*. 

I An equilibrium point x* is said to be asymptotically stable if it is also 
convergent, where convergence requires the existence of a positive such 
5 that 

||x(0)-x*|| < 5 =^> lim ^_ >oo x(0 = x*. (2.6.2) 
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2. 6. Stability 

l If the equilibrium point is convergent, the trajectory can be made approaching to 
x* as t goes to infinity, by choosing again an initial State x(0) close enough 
to x* 

l Notice that asymptotically stable States correspond to attractors of the system. 

I For an autonomous nonlinear dynamic system the asymptotic stability of an 
equilibrium x* can be decided by the existence of energy functions, which 
are also called Liapunov functions. 
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2. 6. Stability Liapunov Function 

■ A continuous function L(x) with a continuous time derivative L'(x)=dL(x)/ck is a 
definite Liapunov function if it satisfies: 

a) L(x) is bounded 


b) L'(x) is negative definite, that is: 

L'(x)<0forx^x* 

(2.6.3) 

L'(x) = 0 for x = x* 

(2.6.4) 

condition (2.6.3) is in the form 

/(x) < 0 for x x * 

(2.6.5) 


the Liapunov function is called semidefinite. 
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2.6. Stability Liapunov' s Theorem 

i The stability of an equilibrium point can be decided by using the following 
theorem: 

The equilibrium State x* is stable (asymptotically stable), if there exists a 
semidefinite (definite) Liapunov function in a small neighborhood of x* 

l The use of Liapunov functions makes it possible to decide the stability of 
equilibrium points without solving the state-space equation of the system. 

• Unfortunately there is not a formal way to find a Liapunov function, mostly it is 
determined in a trial and error fashion. If we are able to find a Liapunov 
function, then we State the stability of the system. However, the inability to 
find a Liapunov function, does not imply the instability of the system. 

I Often convergence of neural networks is guaranteed by an introduction of an 
energy function together with the network itself. 
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2.6. Stability Liapunov' s Theorem 

■ In faet the energy funetions are Liapunov funetions, so non-inereasing along 
trajectories. Therefore the dynamics of the network can be visualized in 
terms of some multidimensional 'energy landscapes' as given previously 
in Figure 2.3. 

I The attractors of the dynamic system are the local minima of the energy funetion 
surrounded with 'valleys' corresponding to the basins of attraction (Figure 



Figure 2.4. Energy landscape and basin attractions 
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2. 7. Effect of input and initial State on the attraction 

l The convergence of a network to an attractor of the activation dynamics may be 
viewed as a retrieval process in which the fixed point is interpreted as the 
output of the neural network. 

l As an example consider the following network dynamic: 

Y t x i (0 = ~ x i (0 + fi (Z w ji x j + 6 i ) (2.7.1) 

Assume that the weight matrix W is fixed and the network is specified 
through 0 and initial State x(0). Both 0 and x(0) are ways of introducing an 
input pattern into the network, although they play distinet dynamical roles 
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2. 7. Effect of input and initial State on the attraction 

Remember 

^ X,. (0 = -X,. (t) + f (£ Wj,Xj +6,) (2.7.1) 

i We then distinguish two modes of operation, depending on whether 
■ network has fixed x(0) and input is applied as 0=u 
i network has fixed 0 and x(0)= u is chosen. 
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2. 7. Effect of input and initial State on the attraction 



Figure 2.5. a) The same initial value 
x(0) may result in different fixed 
points as final value for different u 


case 1: network has fixed x(0) and input is 
applied as 0=u 

The vector u acts as input and the initial is 
State set to some constant vector for all 
inputs. In general, the value of the 
attractors vary smoothly as the vector u 
is varied, hence the network provides a 
continuous mapping between the input 
and the output spaces (Figure 2.5.a). 
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2. 7. Effect of input and initial State on the attraction 

Case 2: network has fixed 0 and x(0)= u is 
chosen. 

In this case, the input pattern is presented to the 
network through the initial State x(0) while having 
fixed 0. The attractors of the dynamics may be 
used to represent items in a memory while the 
initial States are the stimulus to remember the 
stored memory items. The initial States that 
contain incomplete or erronous information may be 
considered as queries to the memory. The network 
then converges to the complete memory items that 
best fits the stimulus. (Figure 2.5. b) 



Figure 2.5. b) Different 
x(0) may converge to 
different fixed values 
although u is the same 
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2.8 Cohen-Grossberg Theorem 

Cohen-Grossberg theorem is useful in deciding the stability of a certain class of 
neural networks. 

Theorem: Given a neural network with N processing elements having bounded 
output signals and transfer functions of the form 

~ a i = a i(.a i )(P i (a i )-Y J Wjifj(a j )) i = l.N (2.8.1 ) 

7=1 

satisfying constraints: 

a) Symmetry: 

w ji= w ij i,j = l-N (2.8.2) 
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2.8 Cohen-Grossberg Theorem 

b) Nonnegativity: 

a t (a) > 0 i = l..N (2.8.3) 

c) Monotonocity: 

= 0 for a> 0 (2.8.4) 

da 

Then the network will converge to some stable point and there will be at most a 
countable number of such stable points. 

The function 

£ P, O')/ AM? (2.8.5) 

*= 1 /=! M o 

is an energy function of the system. That is, E has negative time derivative on 
every possible trajectory that the network's State can follow. 
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2.8 Cohen-Grossberg Theorem 
Proof: 

Due to condition (1) W is symmetric. The time derivative of the energy 
function can be written as 

dF N N 

— = «, («,■ )f'Mi XA («, ) - X w jifj ( a j )) 2 (2.8.6) 

dt tt tt 

and it has negative value for a^ a* whenever conditions (2) and (3) are 
satisfied. 

Since 

— — < 0 for a.^a* (2.8.7) 

dt 1 1 V 1 

the global system is therefore asymptotically stable. 
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2. 9 Hopfield NetWork 


• The continuous deterministic Hopfield model which is based on continuous 
variables and responses, is proposed in [Hopfield 84] to extend their discrete- 
memory model [Hopfield 82] for the processing elements to resemble actual 
neurons more closely. 

• In this model the neurons are modeled as amplifiers in conjunction with 
feedback circuits made up of wires, resistors and capacitors which suggests 
the possibility of building these circuits using VLSI technology (Figure 2.6). 
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2. 9 Hopfield NetWork 




Figure 2.6 Hopfield Network made of electronical components 
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2. 9 Hopfield NetWork 


The output of the amplifier, xi, is a continuous, monotonically increasing 
function of the instantaneous input ai to the /th amplifier. The input-output 
relation of the / 1h amplifier is given by 


fi(a) = tanh(A-a) 


( 2 . 9 . 1 ) 


where k, is a constant called the gain parameter. 
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2. 9 Hopfield NetWork 


Notice that since 

tanh(x) = 6 6 (2.9.2) 

e x +e 

the amplifier transfer function is in faet a sigmoid function 


/<(«) 


i - KM 

l-e 1 

i , -K.a 

\ + e 1 


1 + e 


-2 aKj 


-1 


( 2 . 9 . 3 ) 


as given in equation (1 .2.8) with k -2 k, but shifted so that to have values 
between -1 and +1. 
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2. 9 Hopfield NetWork 

In Figure 2.7, the transfer function is illustrated for several values of k. This 
function is differentiable at each point and always has positive derivative. In 
particular, its derivative at origin gives the gain /, that is 


dfi_ i 
da ' a=0 


(2.9.4) 


+1 

0 

• x =f(Ka)=tanb(Ka) 

r 

f 

A 


//j 





Figure 2.7 Output function used in Hopfield network 
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2. 9 Hopfield Network 

The amplifiers in the Hopfield Circuit correspond to the neurons. A set of 
nonlinear differential equations describes the dynamics of the network. 

The input voltage a, of the amplifier / and it is determined by the equation 

c i da '^ = ~di(t) + Yj w j ifj( a j(t)) + øi (2.9.5) 

j 

while 

x i = fj( a i) (2.9.6) 

corresponds to the output voltage. In Eq. (2.9.5) R i is determined as 

1 /R r P i + ZjWji 
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2. 9 Hopfield NetWork 

Remember 

c i da ^ = ~~R i a i : (O + + e i (2.9.5) 

j 

The State of the network is described by an N dimensional State vector where N is 
the number of neurons in the network. The / 1h component of the State vector is 
given by the output value of the / th amplifier taking real values between -1 and 1 . 
The State of the network moves in the State space in a direction determined by 
the above nonlinear dynamic equation (2.9.5). 
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2. 9 Hopfield Network 

Remember 

c i = - T a ' W + Z w ji fj ( a j (0) + °1 (2.9.5) 

j 

Given the neuron characteristics by (2.9.5), Hopfield network can be represented 
by a neural network as shown in Figure 2.8. 



Figure 2.8 Hopfield Network made of neurons 
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2. 9 Hopfield NetWork 

The energy function for the continuous Hopfield model is given by the formula 

£ = -tZ Z w AA+ZiJ Z^-Jc,- (2.9.7) 

i j i i 

where f . 4 is the inverse of the function fi, that is 

f~\x i ) = a i (2.9.8) 
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2. 9 Hopfield NetWork 

Remember 


In particular, for the transfer function defined by the equation (2.9.3), we have 
= - ln^-A (2.9.9) 

' 1 + JC 

which is shown in Figure 2.9. 
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2.9 Hopfield NetWork : Stability 

One way to show the stabilty of Hopfield network is to show that its energy 
function is a Liapunov function. 

For energy E of the Hopfield network to be a Lyapunov function, it should 
satisfy the following constraints: 

a) E(x) is bounded 

b > ^<„ 

dt 
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2.9 Hopfield NetWork: stability 

Remember: 

£ = -tZ H w ji x j x i+ ZiJ *‘fi~ 1 ( x ) dx -'Z 0 i x i 

i j i i 

Because the function tanh(a) is used in the system as the output function, it limits 
the State variable to take value between -1<x.<1. Furthermore, because the 
integral of the inverse of this function is bounded if -1<x.<1, the energy function 
given by Eq. (2.9.8) is bounded. 
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2.9 Hopfield NetWork: stability 

It can be easily shown that the derivative of the energy function is equivalent 
to: 

dE = y c 1 (*) <&,- 2 (2.9.15) 

dt . 1 dx dt 

i 

Due to equation (2.9.9) we have 

(2-9.16) 

dx 


for any value of x. So Eq. (2.9.15) implies that, 

^<0 

dt 


(2.9.17) 
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2.9 Hopfield NetWork: stability 

Remember 

Yj w ji x j x i + 'Li\ ]fi~\ x ) dx -'L e i x i 

i j i i 

Therefore the energy function described by equation (2.9.8) is a Lyapunov 
function for the Hopfield network when the connection weights are 
symmetrical. 

This means that, whatever the initial State of the network is, it will converge 
to one of the equilibrium States depending on the basin attraction in which 
the initial State lies. 
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2.9 Hopfield Network: stability using C-G theorem 

Remember: 

C i ^dP~ = ~~Rt a i^ + Yj W jifj ( a j (0) + fy 


Another way to show that the Hopfield network is stable is to apply the 
Cohen-Grossberg theorem given in section 2.8. For this purpose we 
reorganize the Eq. (2.9.5) as: 

= c ( ET a 40 + ^)-Z (_ N')/7 (a / ( 0)) (2.9.18) 
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d n 

-a i =a i (a i )(P i (a i )-Y J Wj i fj{aj)) i = \..N (2.8.1) 

7=1 

= A *7 a i ( f ) + Ø i ) - Z (-' w ji )fj ( a j (0)) 


If we compare Eq. (2.9.18) with Eq. (2.8.2) we recognize that in faet Hopfield 
network is a special case of the system defined in Cohen-Grossberg theorem: 



w ij«- w ij 

(2.9.19) 

and 

a i( a i) -pr 

(2.9.20) 

and 

„s . xx CLi(t) 



D + fy 

K i 

(2.9.21) 
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2. 9 Hopfield Network 


It satisfies the conditions on 
a) symmetry because implies 


b) nonnegativity because 

«,■(«,■) = p >0 

i 

c) monotonocity because of 

f\a ) = — tanh(m) > 0 
dt 


(2.9.22) 


(2.9.23) 


(2.9.24) 
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2. 9 Hopfield NetWork 


Therefore, according to the Cohen-Grossberg theorem, the energy function 
defined as 

£ = /ZZ (-%)/(«,)/K)-Z J ](r^a + e t )f\a)da (2.9.25) 

i j i 

is a Lyapunov function of the Hopfield network and the network is globally 
asymptotically stable. 
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2. 9 Hopfield Network 

Remember: 

£ = Z w AA + ZiJ ]fr\x)dx-J j e i x, 

i j i i 

£ = /ZZ ("W#)/(a, )/(«;)- Z I ]{-±a + e t )na)da 

i j i 

In faet, the energy equation defined by equation (2.2.25) can be easily 
reorganized as the one given in equation (2.9.7) (see leeture notes) 


(2.9.7) 

(2.9.25) 
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2. 9 Hopfield NetWork 

As the time derivative of the Energy function is 
negative, the change in the State value of the 
network is in a direction where the energy 
decreases. 

The behavior of a Hopfield network of two 
neurons is demonstrated in the Figure 2.10 
[Hopfield 84]. In the figure the ordinate and 
absisca are the outputs of each neuron. The 
network has two stable States and they are 
located near the upper left and lower right 
corners. 



Figure 2.10 Energy contour map 
for a two neuron two stable 
system 
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2. 9 Hopfield Network 

The second term of the energy function in Eq. (2.9.7), which is 

t ' N lf-'(x)dx (2.9.30) 

alters the energy landscape. 

The value of the gain parameter determines how close the stable points come 
to the hypercube corners. 

In the limit of very high gain, k^oo, this term approaches to zero and the stable 
points of the system lie just at the corners of the Hamming hypercupe 
where the value of each State component is either -1 or 1 . 

For finite gain, the stable points move towards the interior of the hypercube. As 
the gain becomes smaller, these stable points gets doser. When k=0, 
only a single stable point exists for the system Therefore the choice of the 
gain parameter is quite important for the success of the operation 
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